Photonic communication platform

ABSTRACT

Described herein are photonic communication platforms that can overcome the memory bottleneck problem, thereby enabling scaling of memory capacity and bandwidth well beyond what is possible with conventional computing systems. Some embodiments provide photonic communication platforms that involve use of photonic modules. Each photonic module includes programmable photonic circuits for placing the module in optical communication with other modules based on the needs of a particular application. The architecture developed by the inventors relies on the use of common photomask sets (or at least one common photomask) to fabricate multiple photonic modules in a single wafer. Photonic modules in multiple wafers can be linked together into a communication platform using optical or electronic means.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application Ser. No. 62/961,448, entitled “PHOTONICSCOMMUNICATIONS PLATFORM WITH SINGLE LITHOGRAPHIC MASK SET,” filed onJan. 15, 2020 under Attorney Docket No. L0858.70013US02, which is herebyincorporated herein by reference in its entirety.

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application Ser. No. 62/923,889, entitled “PHOTONICSCOMMUNICATIONS PLATFORM WITH SINGLE LITHOGRAPHIC MASK SET,” filed onOct. 21, 2019 under Attorney Docket No. L0858.70013US01, which is herebyincorporated herein by reference in its entirety.

This application claims the benefit under 35 U.S.C. § 119(e) of U.S.Provisional Patent Application Ser. No. 62/814,444, entitled “PHOTONICSCOMMUNICATIONS PLATFORM WITH LITHOGRAPHIC MASK SET,” filed on Mar. 6,2019 under Attorney Docket No. L0858.70013US00, which is herebyincorporated herein by reference in its entirety.

BACKGROUND

Computer systems include random-access memories (RAM) for storing dataand machine code. RAMs are typically volatile memories, such that thestored information is lost when power is removed. In modernimplementations, memories take the form of integrated circuits. Eachintegrated circuit includes several memory cells. To enable access tostored data and machine code, memories are place in electricalcommunication with processors. Typically, these electricalcommunications are implemented as metal traces formed on the substrateson which the memories and the processors are disposed.

SUMMARY OF THE DISCLOSURE

Some embodiments relate to a photonic system comprising a plurality ofphotonic modules including at least first and second photonic modulespatterned according to at least one common photomask. Each of the firstand second photonic modules comprises first and second boundaries; anoptical distribution network; a first optical waveguide opticallycoupling the optical distribution network to a first neighboringphotonic module of the plurality of photonic modules, the firstneighboring photonic module being adjacent to the first boundary; and asecond optical waveguide optically coupling the optical distributionnetwork to a second neighboring photonic module of the plurality ofphotonic modules, the second neighboring photonic module being adjacentto the second boundary.

In some embodiments, the first and second boundaries are opposite to oneanother.

In some embodiments, the first and second optical waveguides arepatterned according to the at least one common photomask.

In some embodiments, each of the first and second photonic modulesfurther comprises an out-of-plane optical coupler optically coupled tothe optical distribution network.

In some embodiments, the optical distribution network is configured toselectively place the first neighboring photonic module in opticalcommunication with the second neighboring photonic module.

In some embodiments, each of the first and second photonic modules ispatterned according to a common set of photomasks, wherein the at leastone common photomask is part of the common set of photomasks.

In some embodiments, each of the first and second photonic modulesfurther comprises: third and fourth boundaries, wherein the first andsecond boundaries are opposite to one another and the third and fourthboundaries are opposite to one another; a third optical waveguideoptically coupling the optical distribution network to a thirdneighboring photonic module of the plurality of photonic modules, thethird neighboring photonic module being adjacent to the third boundary;and a fourth optical waveguide optically coupling the opticaldistribution network to a fourth neighboring photonic module of theplurality of photonic modules, the fourth neighboring photonic modulebeing adjacent to the fourth boundary.

In some embodiments, the optical distribution network is configured toselectively place the first neighboring photonic module in opticalcommunication with the second neighboring photonic module or the thirdneighboring photonic module.

In some embodiments, the optical distribution network comprises aplurality of optical switches.

In some embodiments, the first and second photonic modules are adjacentto one another such that the second photonic module is the firstneighboring photonic module for the first photonic module.

Some embodiments relate to a method for manufacturing a semiconductorwafer comprising: patterning each of at least some of a plurality ofphotonic modules on the semiconductor wafer using at least one commonphotomask, wherein patterning each of at least some of the plurality ofphotonic modules comprises: patterning an optical distribution network;patterning a first optical waveguide optically coupling the opticaldistribution network to a first neighboring photonic module of theplurality of photonic modules, the first neighboring photonic modulebeing adjacent to a first boundary of the photonic module; andpatterning a second optical waveguide optically coupling the opticaldistribution network to a second neighboring photonic module of theplurality of photonic modules, the second neighboring photonic modulebeing adjacent to a second boundary of the photonic module.

In some embodiments, patterning each of at least some of the pluralityof photonic modules comprises patterning the first and second opticalwaveguides using the at least one common photomask.

In some embodiments, the method further comprises dicing thesemiconductor wafer to obtain a photonic substrate comprising: a firstphotonic module of the plurality of photonic modules; a firstneighboring photonic module adjacent to the first boundary of the firstphotonic module; and a second neighboring photonic module adjacent tothe second boundary of the first photonic module.

In some embodiments, the first and second boundaries of the firstphotonic module are opposite to one another.

In some embodiments, patterning each of at the at least some of theplurality of photonic modules further comprises: patterning a thirdoptical waveguide optically coupling the optical distribution network toa third neighboring photonic module of the plurality of photonicmodules, the third neighboring photonic module being adjacent to a thirdboundary of the photonic module; and patterning a fourth opticalwaveguide optically coupling the optical distribution network to afourth neighboring photonic module of the plurality of photonic modules,the fourth neighboring photonic module being adjacent to a fourthboundary of the photonic module. The first and second boundaries areopposite to one another and the third and fourth boundaries are oppositeto one another.

In some embodiments, patterning each of at least some of the pluralityof photonic modules further comprises: patterning a first photonicmodule using a first photolithography shot in connection with the atleast one common photomask; and patterning a second photonic moduleusing a second photolithography shot, subsequent to the firstphotolithography shot, in connection with the at least one commonphotomask.

Some embodiments relate to a computing system comprising a photonicsubstrate patterned with a plurality of photonic modules including atleast first and second photonic modules, each of the first and secondphotonic modules being patterned according to at least one commonphotomask, wherein the first photonic module is optically coupled to thesecond photonic module; a first die in communication with the firstphotonic module; and a second die in communication with the secondphotonic module.

In some embodiments, the first die comprises a processor and the seconddie comprises a memory.

In some embodiments, the computing system further comprises a laser diecoupled to the photonic substrate.

In some embodiments, each of the first and second photonic modulescomprises: first and second boundaries; an optical distribution network;a first optical waveguide optically coupling the optical distributionnetwork to a first neighboring photonic module of the plurality ofphotonic modules, the first neighboring photonic module being adjacentto the first boundary; and a second optical waveguide optically couplingthe optical distribution network to a second neighboring photonic moduleof the plurality of photonic modules, the second neighboring photonicmodule being adjacent to the second boundary.

In some embodiments, the first and second boundaries are opposite to oneanother.

In some embodiments, the first and second optical waveguides arepatterned according to the at least one common photomask.

In some embodiments, each of the first and second photonic modulecomprises an out-of-plane optical coupler, wherein: the first die isoptically coupled to the out-of-plane optical coupler of the firstphotonic module, and the second die is optically coupled to theout-of-plane optical coupler of the second photonic module.

In some embodiments, the first die is coupled to a first side of thephotonic substrate and the second die is coupled to a second side of thephotonic substrate opposite the first side.

In some embodiments, the computing system further comprises a third diestacked on top of the first die.

In some embodiments, the first and second photonic modules share aboundary such that the first photonic module is adjacent to the secondphotonic module.

In some embodiments, the first die is mounted above or below the firstphotonic module; and the second die is mounted above or below the secondphotonic module.

In some embodiments, the first die is in electronic communication withthe first photonic module, and the second die is in electroniccommunication with the second photonic module.

Some embodiments relate to a multi-node computing system comprising: aplurality of computing systems including at least first and secondcomputing systems, each of the first and second computing systemscomprising: a photonic substrate patterned with a plurality of photonicmodules including at least first and second photonic modules, each ofthe first and second photonic modules being patterned according to atleast one common photomask, wherein the first photonic module isoptically coupled to the second photonic module; a first die incommunication with the first photonic module; a second die incommunication with the second photonic module; and a fiber connectingthe first and second computing systems to one another.

In some embodiments, each of the first and second computing systemsfurther comprises a fiber coupler, wherein the fiber optically couplesrespective fiber couplers of the first and second computing systems toone another.

In some embodiments, the first die comprises a processor and the seconddie comprises a memory.

In some embodiments, each of the first and second computing systemsfurther comprises a laser coupled to the photonic substrate.

In some embodiments, each of the first and second photonic modules ofthe photonic substrate comprises: first and second boundaries; anoptical distribution network; a first optical waveguide opticallycoupling the optical distribution network to a first neighboringphotonic module of the plurality of photonic modules, the firstneighboring photonic module being adjacent to the first boundary; and asecond optical waveguide optically coupling the optical distributionnetwork to a second neighboring photonic module of the plurality ofphotonic modules, the second neighboring photonic module being adjacentto the second boundary.

In some embodiments, the first and second optical waveguides arepatterned according to the at least one common photomask.

In some embodiments, each of the first and second photonic modulescomprises an out-of-plane optical coupler, wherein: the first die isoptically coupled to the out-of-plane optical coupler of the firstphotonic module, and the second die is optically coupled to theout-of-plane optical coupler of the second photonic module.

In some embodiments, the first die is coupled to a first side of thephotonic substrate and the second die is coupled to a second side of thephotonic substrate opposite the first side.

In some embodiments, the multi-node computing system further comprises athird die stacked on top of the first die.

In some embodiments, the first and second photonic modules share aboundary such that the first photonic module is adjacent to the secondphotonic module.

In some embodiments, the first die is mounted above or below the firstphotonic module; and the second die is mounted above or below the secondphotonic module.

Some embodiments relate to a photonic communication platform comprising:a photonic network comprising a plurality of optical switches formed ona semiconductor substrate; a plurality of dies in communication with thephotonic network; an electronic switching network comprising a pluralityof transistors co-integrated with the plurality of optical switches, theelectronic switching network being configured to: at a first time,program the optical switches to form a first optical communication pathcoupling together a first subset of the plurality of dies, and at asecond time subsequent to the first time, program the optical switchesto form a second optical communication path coupling together a secondsubset of the plurality of dies, the second optical communication pathbeing distinct from the first communication path.

In some embodiments, the plurality of transistors are formed on thesemiconductor substrate.

In some embodiments, the semiconductor substrate is a firstsemiconductor substrate, and wherein the plurality of transistors areformed on a second semiconductor substrate, wherein the first and secondsemiconductor substrates are 3D-bonded together.

In some embodiments, programming the optical switches to form a firstoptical communication path comprises: identifying an opticalcommunication path coupling together first subset of the plurality ofdies; and programming the optical switches based on the identifiedoptical communication path.

In some embodiments, identifying an optical communication path couplingtogether first subset of the plurality of dies comprises monitoring ausage of the photonic network.

In some embodiments, the electronic switching network is furtherconfigured to determine at least one characteristic of an optical signalat the first optical communication path; identify an encoding schemebased on the at least one characteristic of the optical signal; andcause the photonic network to communicate optically on the first opticalcommunication path based on the encoding scheme.

In some embodiments, the plurality of dies are in electroniccommunication with the photonic network.

In some embodiments, the electronic switching network is furtherconfigured to cause the photonic network to communicate optically on thefirst optical communication path using wavelength division multiplexing.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and embodiments of the application will be describedwith reference to the following figures. It should be appreciated thatthe figures are not necessarily drawn to scale. Items appearing inmultiple figures are indicated by the same reference number in thefigures in which they appear.

FIG. 1 illustrates a computing system based on a photonic communicationplatform, in accordance with some embodiments.

FIG. 2A illustrates a semiconductor wafer, in accordance with someembodiments.

FIG. 2B illustrates a set of photomasks, in accordance with someembodiments.

FIG. 2C illustrates an example photomask for forming optical waveguides,in accordance with some embodiments.

FIG. 2D illustrates the wafer of FIG. 2A patterned in accordance withthe photomask set of FIG. 2B, in accordance with some embodiments.

FIG. 2E identifies photonic substrates formed on the patterned wafer ofFIG. 2D, in accordance with some embodiments.

FIG. 3A illustrates an example photonic module of the patterned wafer ofFIG. 2E, in accordance with some embodiments.

FIG. 3B illustrates optical out-of-plane coupling between a photonicmodule and a die, in accordance with some embodiments.

FIG. 3C illustrates optical out-of-plane coupling between photoniccommunication fabrics, in accordance with some embodiments.

FIG. 3D illustrates a group of photonic modules of the type illustratedin FIG. 3A, in accordance with some embodiments.

FIG. 3E illustrates a waveguide arrangement at the boundary between apair of adjacent photonic modules, in accordance with some embodiments.

FIG. 3F illustrates another waveguide arrangement at the boundarybetween a pair of adjacent photonic modules, in accordance with someembodiments.

FIG. 3G illustrates another waveguide arrangement at the boundarybetween a pair of adjacent photonic modules, in accordance with someembodiments.

FIG. 3H illustrates a group of photonic modules sharing the same patternof metal traces, in accordance with some embodiments.

FIG. 3I illustrates a cross section of a photonic substrate, inaccordance with some embodiments.

FIG. 4 illustrates a photonic system including dies mounted to aphotonic substrate, in accordance with some embodiments.

FIG. 5A illustrates an example optical distribution network, inaccordance with some embodiments.

FIG. 5B illustrates a plurality of photonic modules each including theoptical distribution network of FIG. 5A, in accordance with someembodiments.

FIG. 5C illustrates another example optical distribution network, inaccordance with some embodiments.

FIG. 5D illustrates another example optical distribution network, inaccordance with some embodiments.

FIG. 6A illustrates a photonic module including a transceiver and aplurality of switches, in accordance with some embodiments.

FIG. 6B illustrates the transceiver of FIG. 6A in additional detail, inaccordance with some embodiments.

FIG. 6C illustrates a plurality of photonic modules of the typeillustrated in FIG. 6A, in accordance with some embodiments.

FIG. 7A illustrates a photonic communication platform including anelectronic switching network, in accordance with some embodiments.

FIG. 7B illustrates the photonic communication platform of FIG. 7Acontrolled to form a first optical path, in accordance with someembodiments.

FIG. 7C illustrates the photonic communication platform of FIG. 7Acontrolled to form a second optical path, in accordance with someembodiments.

FIG. 8A illustrates a computing system based on a photonic communicationplatform, in accordance with some embodiments.

FIG. 8B illustrates a multi-node computing system including a pluralityof computing systems of the type illustrated in FIG. 8A, in accordancewith some embodiments.

DETAILED DESCRIPTION I. Scalability of Memory Bandwidth

The inventors have recognized and appreciated that one of the majorbottlenecks limiting the spread of data-intensive computing is theinability to scale memory capacity and bandwidth in modern computers atsufficiently high rates. The inventors have developed photoniccommunication platforms that enable scaling of memory capacity andbandwidth well beyond what is possible with conventional computers.

Data-intensive computing systems, such as those designed to handle deeplearning algorithms, require access to vast amounts of data, thusincreasing the requirements for memory capacity. Further, mostapplications require access to data in real time or quasi real time,thus increasing the requirements for memory bandwidth. Some conventionalcomputing systems utilize graphical processing units (GPUs) to improvememory access efficiency. Some GPUs transfer data from a memory atbandwidths as high as 256 GB/s. While such a memory bandwidth may besufficient for most graphic-based applications, this is far from beingenough for data-intensive applications such as deep neural networks andhigh-frequency trading.

Deep neural networks rely on large numbers of parameters such as weightsand activation parameters. A typical 50-layer network with 26 millionweight parameters, for example, can compute up to 16 million activationsin a forward pass. If weight and activation parameters are stored using32-bit floating point values, the total storage requirement is 168 MB.In addition, if data are represented as dense vectors, the memoryrequirement can increase to several gigabytes. During training, localityof the training dataset is important because of the frequency of accessto these large datasets. These amounts of data are far too large to bestored in the internal memory of the GPU, thereby requiring use ofseveral external dynamic random access memories (DRAMs). Each DRAM chipcommunicates with a processor through electronic communication carriers.For example, in computing systems that utilize silicon interposers, inwhich the processor and the memory chips are mounted on the sameinterposer, memory-processor communication is supported by conductivetraces formed on the interposer. Use of silicon interposers has becomewidespread in recent years because the densities of conductive tracesthat an interposer can provide is much larger than in printed circuitboards (PCBs). However, interposers cannot be scaled indefinitely.Interposers, being manufactured using microfabrication techniques, havelimited areas, thus limiting the numbers of memory chips that aninterposer can accommodate.

Moreover, the presence of parasitic impedance further limits thebandwidth scalability. Ideally, the impedance of the interposer'sconductive traces would be negligible. In reality, unfortunately, theimpedance of the traces is significant. The parasitic impedance limitsbandwidth scalability in two ways. First, it limits the bandwidth that atrace can support. Second, it increases power consumption. To makethings worse, parasitic impedance increases with the length of a trace,meaning that the larger the separation between a memory chip and aprocessor, the lower the bandwidth. This is why conventional computingsystems are typically designed so that memory chips are positionedwithin a few centimeters of the processor. However, there are only somany memory chips that can be accommodated within this range. The resultis that conventional computing systems are limited in both memorybandwidth and memory capacity.

II. Overview of Photonic Communication Platforms

The communication platforms developed by the inventors overcome theselimitations using photonics. The physics according to which lightpropagates inside a waveguide makes optical communications inherentlyimmune to parasitic impedance. The immunity to parasitic impedance leadsto a major benefit—it removes the requirement that memory chips bepositioned within a certain range of the processor.

Another advantage of the photonic communication platforms developed bythe inventors is that they can be easily adapted to different computerarchitectures. Single-node computer architectures involve one processorchip, which may have more than one processor core per chip, and severalmemory chips. Multi-node computer architectures involve severalprocessor chips and several memory chips. Some multi-node architecturesuse a ring topology—each processor is in direct communication with twoneighboring processors and communication with other processors passesthrough the neighboring processors. Other multi-node architectures use astar topology—a central hub is responsible for routing core-to-corecommunications. Yet another multi-node architectures use a multi-casttopology—each processor is in directed communication to several otherprocessors.

Some aspects of the photonic communication platforms described hereinmake them easily adaptable to any one of these architectures (andothers). Some embodiments provide photonic communication platforms thatuse “photonic modules.” Each photonic module includes programmablephotonic circuits that can be configured based on the needs of aparticular computer architecture. Some platforms are arranged accordingto 1-dimensional schemes, such as in blocks of 3×1 modules, in blocks of5×1 modules, in blocks of 10×1 modules, 20×1 modules, etc. Someplatforms are arranged according to 2-dimensional schemes, such as inblocks of 3×3 modules, in blocks of 5×3 modules, in blocks of 5×5modules, in blocks of 10×10 modules, etc. More generally, the platformsenable any block of N×M modules, where N≥1 and M≥1, and any topology,such as T-topologies, L-topologies, X-topologies, etc. Each photonicmodule can serve as a node of a computing system. At each node, theremay be one or more digital processor chips, one or more analogaccelerators, one or more photonic accelerators, one or more memorychips, one or more networking chips, or other devices.

FIG. 1 illustrates an example computing system based on a photoniccommunication platform with nine photonic modules arranged in a 3×3topology, in accordance with one example. Computing system 10 includes aphotonic substrate 20 patterned with nine photonic modules 22 (alsoreferred to herein as “photonic sites,” or simply “sites”). Thisphotonic communication platform supports one processor die (30)positioned in the middle of photonic substrate 20, and eight memorynodes surrounding the processor die. Some of the memory nodes include asingle memory chip (see for example memory die 32). Other memory nodesinclude a stacked memory including multiple vertically-stacked memorydies (see for example stacked memory 34). The dies can communicate withthe photonic module electronically (e.g., using through-silicon vias,copper pillars, micro-bumps, ball-grid arrays or other electricalinterconnects) and/or optically (e.g., using grating couplers, prisms,lenses or other optical couplers).

As described in detail further below, the photonic modules are patternedwith optical waveguides and optical distribution networks. The opticaldistribution network of a photonic module can selectively place the dieof that particular photonic module in optical communication with anyother die of the computing system. For example, the optical distributionnetwork of the photonic module positioned under processor die 30 may bereconfigured depending on the needs of the processor. At the beginningof a routine, the processor may need to access data stored in a firstmemory node. This read operation involves configuring the respectiveoptical distribution networks to place the processor in opticalcommunication with the first memory node. Later in the routine, theprocessor may need to write data into a second memory node. This writeoperation involves reconfiguring the optical distribution networks toplace the processor in optical communication with the second memorynode.

The inventors have further appreciated that manufacturing photonicmodules in large scales can be costly. The photonic communicationplatforms described herein are engineered in a way that limitsmanufacturing costs. These platforms rely on the use of common photomasksets (or at least one common photomask) to fabricate multiple photonicmodules. This approach reduces costs in two ways. First, it reducesadditional costs that would otherwise be incurred in procuring severaldifferent photomask sets. Second, it enables fabrication of photonicmodules using standard semiconductor foundries, some of which requirethat the same photomask set (or at least one photomask) be used acrossan entire wafer. Designing photonic modules that share at least onephotomask enables fabrication of many photonic modules on the samesemiconductor wafer while leveraging standard, low-cost step-and-repeatmanufacturing processes.

III. Photonic Modules

The photonic modules described herein may be manufactured usingmicrofabrication techniques, including for example complementarymetal-oxide-semiconductor (CMOS) microfabrication techniques.Accordingly, some embodiments relate to silicon photonics-based opticalcommunication platforms. Some particular microfabrication techniquesinvolve step-and-repeat approaches—whereby stepper machines are used topattern a semiconductor wafer with multiple copies of a template layout.FIG. 2A through FIG. 2E illustrate microfabrication techniques formanufacturing photonic modules. FIGS. 3A through 3F illustrate examplesof photonic modules patterned using these microfabrication techniques.

Referring first to FIG. 2A, this figure illustrates a semiconductorwafer 100. Wafer 100 may be made of any material. For example, wafer 100may be made of (or otherwise include) silicon. In one example, wafer 100is a silicon-on-insulator (SOI) wafer. In another example, wafer 100 isa bulk silicon wafer. Wafer 100 may have any size. For example, thediameter of wafer 100 may be 150 mm, 300 nm, or 450 mm, among otherpossible values. However, not all wafers need to have a circular shape.

FIG. 2B illustrates a set of photomasks that may be used for patterningwafer 100 using photolithographic techniques. Photomask set 200 includesthree photomasks (201, 202 and 203), though other sets may include moreor fewer photomasks. Each photomask has a particular pattern of opaqueand transparent regions. When the photomask is exposed to light, theopaque regions block the light, thereby preventing it from shining awafer, and the transparent regions allow passage of the light. Theresult is that the pattern of the photomask is transferred to the wafer.

Each photomask may define a particular layer of a photonic module. Onephotomask may be used to define optical waveguides. When the wafer goesthrough an etch process, only the exposed regions (or only thenon-exposed regions) are etched away, while the other regions remainun-etched. This photomask may be patterned to form a network of opticalwaveguides when the wafer is exposed to light through this photomask.FIG. 2C illustrates a portion of a photomask that may be used to formwaveguides on wafer 100. The lines of photomask 201 represent opaqueregions. The background of photomask 201 is transparent. Exposure ofphotomask 201 to light so that an image of the photomask is projectedonto wafer 100 enables patterning of waveguides in the shapes of theopaque regions. In this particular example, the pattern of lines of thephotomask results in a grid of waveguides.

Some photonic modules involve use of different levels of opticalwaveguides. In some such embodiments, photomask set 200 may include adedicated photomask for each waveguide level. Another photomask may beused to define n-doped regions. When the wafer goes through an ionimplantation or dopant diffusion process, only the exposed regions (oronly the non-exposed regions) receive the doping, while the otherregions remain undoped. Another photomask may be used to define p-dopedregions using a similar process. Some photonic modules involve use ofdifferent doping concentrations. In some such embodiments, photomask set200 may include a dedicated photomask for each doping concentration. Inother embodiments, photomask set 200 may include photomasks used todefine deposition of semiconductor materials other than silicon, such asgermanium and/or other materials of the periodic table, such as GroupsIII or V. Another photomask may be used to define metal contacts.Another photomask may be used to define metal traces. Some photonicmodules involve use of different levels of metal traces. In some suchembodiments, photomask set 200 may include a dedicated photomask foreach metal trace level.

In some embodiments, wafer 100 is patterned in a step-and-repeatfashion. When wafer 100 is processed in a stepper machine, the patternof a photomask is exposed repeatedly across the surface of the wafer, ina grid. This process involves moving the wafer in steps back and forthand left and right under the lens of the stepper, and exposing thephotomask at each step. The result is that wafer 100 is patterned withmultiple copies of the pattern defined by a photomask. This operationmay be repeated for each photomask (or at least some photomasks) of theset.

In some embodiments, this process may be used to pattern wafer 100 withmultiple copies of a template photonic module. In the example of FIG.2D, wafer 100 has been patterned with a grid of photonic modules 22. Thephotonic modules may share the pattern of one or more photomasks of set200. For example, the photonic modules may share the pattern of the samewaveguide photomask(s) and/or the same m trace photomask(s). In otherembodiments, the photonic modules share the pattern of all thephotomasks of set 200. For example the photonic modules may share thesame optical waveguide pattern, the same n-doping pattern, the samep-doping pattern, the same contact pattern, the same metal tracepattern, etc.

In some embodiments, the entire surface of wafer 100 is patterned usingphotomask set 200. However, not all embodiments are limited in thisrespect as some portions of wafer 100 may be patterned using a firstphotomask set and other portions of wafer 100 may be patterned using asecond photomask set. In some embodiments, the first and secondphotomask sets may share one or more common photomasks, such as awaveguide photomask.

Once patterned, wafer 100 may include multiple photonic substrates.Photonic modules 22 may be separated from the wafer together to formphotonic substrates of any desired shape and size. For example, thewafer of FIG. 2E has been marked to obtain six photonic substrates fromwafer 100. This figure identifies a 1×1 photonic substrate having onlyone photonic module 22, a 2×2 photonic substrate having four photonicmodules 22, a 2×3 photonic substrate having six photonic modules 22, andthree 3×3 photonic substrates having nine photonic modules 22 each.Separation of a photonic substrate from the wafer involves dicing thewafer along the perimeter of the desired photonic substrate. One of the3×3 photonic substrates of wafer 100 may be used as the photonicsubstrate of the example computing system of FIG. 1 (see photonicsubstrate 20).

The techniques described in connection with FIGS. 2A-2D enablemanufacturing of photonic modules at relatively low costs. Somesemiconductor foundries require that the same photomask set (or at leastone photomask) be used for patterning the entirety of a wafer (or atleast a portion of a wafer). Otherwise, patterning different portions ofa wafer using different photomasks would involve replacing one photomaskfor another between photolithographic exposures, which would make thestep-and-repeat process inefficient and costly. Designing photonicmodules that share at least one photomask enables fabrication of manyphotonic modules on the same semiconductor wafer while leveragingstandard, low-cost step-and-repeat processes.

FIG. 3A illustrates an example photonic module 22. In this example,photonic module 22 is shaped as a rectangle (though other shapes arealso possible, such as squares or other polygons). As such, photonicmodule 22 is bounded by four boundaries (boundaries 1, 2, 3 and 4).Boundary 1 is opposite to boundary 2, and boundary 3 is opposite toboundary 4. Boundary 1 is adjacent to boundaries 3 and 4, and boundary 2is also adjacent to boundaries 3 and 4. Photonic module 22 includes anoptical distribution network 104 coupled to waveguides 111, 112, 113 and114. Waveguide 111 optically couples optical distribution network 104 toboundary 1. As such, optical signals coupled from optical distributionnetwork 104 to waveguide 111 can be transferred outside the photonicmodule by crossing boundary 111. Similarly, waveguide 112 opticallycouples optical distribution network 104 to boundary 2, waveguide 113optically couples optical distribution network 104 to boundary 3 andwaveguide 114 optically couples optical distribution network 104 toboundary 4. In some embodiments, the boundaries of a photonic module aredefined based on a photolithography shot (e.g., the boundaries aredefined by the boundaries of the photomask(s) used to fabricate thephotonic module). In other embodiments, however, one photolithographyshot may define more than one photonic module. For example, a photomaskmay be patterned with multiple side-by-side instances of a templatephotonic module. In some such embodiments, the boundaries of a photonicmodule are defined where adjacent instances of the template photonicmodule meet.

While the example of FIG. 3A illustrates waveguides coupling the opticaldistribution network to each of the boundaries, not all embodiments arearranged in this manner. In other embodiments, a photonic module 22 mayinclude two of these four waveguides, such as waveguides 111 and 112, orwaveguides 111 and 113. In yet other embodiments, a photonic module 22may include three of these four waveguides, such as waveguides 111, 112and 113. Optical distribution network 104 includes photonics components(e.g., photonic switches) for routing optical signals inside and outsidephotonic module 22.

In some embodiments, a photonic module may include multiple layers ofphotonic waveguides. Similar to how multiple layers of conductive tracesincrease an electronic circuit's ability to route electric signals,multiple layers of waveguides increase a photonic module's ability toroute optical signals. In one example, one layer includes siliconwaveguides, and one layer includes silicon nitride waveguides. Inanother example, multiple layers include silicon waveguides.Additionally or alternatively, multiple layers include silicon nitridewaveguides. The choice of material of each waveguide layer may bedetermined by the wavelength of light that will be routed by thewaveguide. For example, silicon and silicon nitride layers may be usedfor routing infrared light in the telecommunication bands withwavelengths around 1.3 μm or 1.5 μm. In some examples, the multiplelayers of waveguides may also include aluminum nitride waveguides thatcan be used to route visible light down to UV wavelengths or aluminumoxide waveguides that are used to route UV light. Each layer may bearranged in a configuration similar to that illustrated in FIG. 3A—withan optical distribution network that routes signals among the waveguidesof the layer.

Photonic module 22 further includes one or more out-of-plane couplers105. Waveguide 117 optically couples out-of-plane coupler 105 to opticaldistribution network 104. Out-of-plane coupler 105 is configured to emitlight received from waveguide 117 outside the xy-plane, for example in adirection parallel to the z-axis or at an angle relative to the z-axis.Out-of-plane coupler 105 may be further configured to capture lightshining from outside the xy-plane and to convey the captured light towaveguide 117. Out-of-plane coupler 105 enables optical communicationbetween photonic module 22 and a die disposed above the photonic moduleand/or below the photonic module. Out-of-plane coupler 105 may beimplemented using any suitable optical component, including for exampleoptical gratings, lenses, and prisms. In some embodiments, the opticaldistribution network may be configured so that the same out-of-planecoupler enables optical communication in both directions—from opticaldistribution network 104 to a die and from the die to opticaldistribution network 104. In other embodiments, one out-of-plane coupler105 may enable optical communication in one direction, and anotherout-of-plane coupler 105 (not shown in FIG. 3A) may enable opticalcommunication in the opposite direction. In one embodiment, theout-of-plane coupler 105 may be used to couple light source into theoptical distribution network 104. The light source can be one of lasers(continuous-wave or pulsed), LEDs, or superluminescent diodes.

FIG. 3B illustrates how an out-of-plane coupler 105 may be used toenable out-of-plane optical communication. For clarity, onlyout-of-plane coupler 105, waveguide 117 and optical distribution network(ODN) 104 are illustrated inside optical module 22. In this example,out-of-plane coupler 105 is implemented with an optical grating. Die 320is mounted to photonic module 22. Die 320 may include a processor, amemory, and/or other electronic components (not illustrated in FIG. 3B).Further, die 320 includes out-of-plane coupler 351, waveguide 317 andcontroller 322. Controller 322 is electrically coupled to opticaldistribution network 104 via electrical connection 324, which mayinclude for example a ball-grid array, copper pillars, through siliconvias, micro-bumps, metals pads, etc. In this example, out-of-coupler 105is emitting light in a direction parallel to the z-axis towardsout-of-plane coupler 351. Out-of-plane coupler captures the light andconveys the captured light to controller 322 via waveguide 317.

Controller 322 controls the operations of optical distribution network104. For example, controller 322 controls the state of the switches ofoptical distribution network 104. Control signals are provided tooptical distribution network 104 via electrical connection 324.Alternatively, or additionally, a controller may be formed directly onphotonic module 22, and this controller may control the operations ofoptical distribution network 104. This controller may provide controlsignals to optical distribution network 22 via conductive traces formedon photonic module 22.

Referring back to FIG. 3A, optical distribution network 104 mayselectively couple any components of photonic module 22 to any othercomponents of photonic module 22. For example, optical distributionnetwork 104 may enable passage of light between waveguide 111 andwaveguide 112, and/or between waveguide 111 and waveguide 113, and/orbetween waveguide 113 and waveguide 114, and/or between out-of-coupler105 and waveguide 111, and/or between out-of-coupler 105 and waveguide113, etc.

FIG. 3C illustrates how an out-of-plane coupler 105 may be used tocommunicate between two photonic communication fabrics. For clarity, thefigure only shows two photonic modules 22, one from each photoniccommunication fabric, optically coupled to one another usingout-of-plane couplers 105. Controller 322 is electrically coupled toboth optical distribution networks 104 using electrical connection 324and the through-silicon-via 125. Having multiple photonic communicationfabrics stacked one on top of another increases the number of opticaland electronic communication channels between each site. Furthermore,having multiple communication fabrics may reduce the number of waveguidecrossings needed to route optical signals across a photonic module, thusreducing optical loss and improving the overall power budget.

A photonic substrate may include multiple photonic modules connectedtogether to collectively form an optical network. FIG. 3D illustrates anexample 2×3 photonic substrate including six photonic modules 22. Thisphotonic substrate is obtained by dicing a group of 2×3 photonic modulesoff of wafer 100 (see FIG. 2E). The photonic modules 22 are arranged tothat waveguide 111 of an optical module is aligned with waveguide 112 ofthe optical module to the left of that optical module, waveguide 112 ofan optical module is aligned with waveguide 111 of the optical module tothe right of that optical module, waveguide 113 of an optical module isaligned with waveguide 114 of the optical module above that opticalmodule and waveguide 114 of an optical module is aligned with waveguide113 of the optical module below that optical module. As a result, theoptical modules form an optical network. Optical distribution networks104 may route optical signals anywhere inside or outside the network.Suppose, for example, that a processor is mounted to the photonic modulepositioned at the north-west corner of the photonic substrate and that amemory is mounted to the photonic module positioned at the south-eastcorner of the photonic substrate. A read operation may involvereconfiguring the optical distribution networks to place the processorin optical communication with the memory. For example, an opticalcommunication path may be formed that 1) couples the processor to theout-of-plane coupler of the photonic module to which the processor ismounted, 2) couples the out-of-plane coupler of that photonic module towaveguide 112 of the same photonic module, 3) couples waveguide 112 ofthat photonic module to waveguide 111 of the adjacent photonic module(mid-uppermost photonic module), 4) couples waveguide 112 of themid-uppermost photonic module to waveguide 111 of the next adjacentphotonic module (north-east corner of the photonic substrate), 5)couples waveguide 114 of the photonic module positioned at north-eastcorner to waveguide 113 of the photonic module to which the memory ismounted, and 6) couples waveguide 113 of the photonic module to whichthe memory is mounted to the out-of-plane coupler of the same photonicmodule.

As discussed above, waveguides of adjacent photonic modules areoptically coupled to one another, thereby permitting passage of lightfrom one photonic module to the next. In some embodiments, thewaveguides may be physically connected. This arrangement is illustratedin FIG. 3E, which depicts the region at the boundary of two adjacentphotonic modules. As shown, waveguide 112 of the photonic modulepositioned on the left-hand side is physically connected to waveguide111 of the photonic module positioned on the right-hand side. In someembodiments, a continuous waveguide crosses the boundary and extendsbetween the respective optical distribution networks of the photonicmodules.

In other embodiments, there may be a gap between the waveguides. Thisarrangement is illustrated in FIG. 3F. In this example, each waveguidehas an end that is located at a distance from the boundary. Thus, a gapis formed at the boundary region. Notwithstanding the gap, waveguides111 and 112 are still optically coupled to each other. In this case, infact, light emitted at the end of a waveguide reaches the end of theother waveguide by free space propagation. If the size of the gap issufficiently small (e.g., less than 500 μm), most of the optical powerradiated by one waveguide couples to the other waveguide.

In yet other embodiments, as illustrated in FIG. 3G, a photonic bridgemay be used to optically couple the waveguides to each other. In thisexample, the ends of the waveguides are coupled to respectiveout-of-plane couplers 152. A photonic bridge die 300 is mounted to theboundary region. Photonic bridge die 300 includes a pair of out-of-planecouplers 352 and an optical waveguide 354 that couples the out-of-planecouplers to one another. Suppose that a processor die 302 needs to senda read-out message to a memory die 304. This can be accomplished by 1)conveying light, out-of-plane, from processor die 302 to the respectivephotonic module (for example in the manner illustrated in FIG. 3B), 2)conveying light to waveguide 112, and consequently, to out-of-coupler152, 3) conveying light to out-of-coupler 352, 4) conveying light towaveguide 354, and consequently, to the other out-of-coupler 352, 5)conveying light to the other out-of-coupler 152, and 6) conveying light,out-of-plane, from that photonic module to the memory die 304 (forexample in the manner illustrated in FIG. 3B).

In some embodiments, photonic modules 22 may be patterned according to acommon metal trace photomask. As a result, the photonic modules sharethe same pattern of metal traces. In some embodiments, photonic modules22 are patterned according to multiple common photomasks. As a result,multiple levels of metal traces share the same patterns across differentphotonic modules. Some of the metal traces may be used to deliver poweracross a photonic substrate. Some of the metal traces may be used todeliver electronic signals across a photonic substrate.

FIG. 3H illustrates a 2×3 photonic substrate, in which each photonicmodule 22 shares the same pattern of metal traces. For the sake ofillustration, only the metal traces are shown in this figure, thougheach photonic module further includes waveguides, one or moreout-of-plane couplers and optical distribution networks, for example inthe arrangement shown in FIG. 3D. In this example, there are two levelsof metal traces. The metal traces of each level has been fabricatedusing the same photomask across the different photonic modules. Themetal traces of metal trace level 1 run in the horizontal direction,thereby electrically coupling photonic modules that are adjacent to oneanother in the horizontal direction. The metal traces of metal tracelevel 2 run in the vertical direction, thereby electrically couplingphotonic modules that are adjacent to one another in the verticaldirection. Of course, other arrangements are also possible. For example,in other embodiments, the metal traces of the same level mayelectrically couple one photonic module to all the photonic modules thatare adjacent to it.

The metal traces are arranged to carry electricity (e.g., signals and/orpower) across the boundaries of the photonic modules. This may beachieved by patterning metal traces to be continuous across theboundaries of the photonic modules. In this example, the metal traces oflevel 1 are continuous across the vertical boundaries, and the metaltraces of level 2 are continuous across the horizontal boundaries. Metaltraces of different levels may be connected to one another using vias(not shown in FIG. 3H). In some embodiments, the photonic modules mayshare the same patterns of vias. In other words, the same via photomaskmay be used for each photonic module. In some embodiments, the photonicmodules may have more (tens to hundreds of) metal traces. Some of thesemetal traces may be arranged to be continuous across photonic modulesbut, in some embodiments, a majority of the metal traces need not bepatterned to be continuous across modules.

Metal traces may be used to deliver power and/or electric signals acrossthe photonic substrate. In one example, a power source is connected toone particular photonic module. Power produced by the power source maybe delivered from that particular photonic module to the other photonicmodules using the metal traces. In another example, a controller chipmay be bonded (e.g., 3D-bonded) to a particular photonic module. Controlsignals produced by the controller may be delivered from that particularphotonic module to the other photonic modules using the metal traces.The control signals may control the states of the optical distributionnetworks of the photonic modules.

As discussed above, electronic control circuits may be used to controlthe operations of the photonic modules. These electronic controlcircuits may control, for example, how optical distribution networks 104route optical signals. The electronic control circuits may beco-integrated with the photonic modules in various ways. In someembodiments, a photonic module may be formed on a first substrate and anelectronic control circuit may be formed on a second substrate. The twosubstrates may bonded together to place the electronic control circuitin electrical communication with the optical distribution networks. Inother embodiments, however, an electronic control circuit may befabricated directly on the same substrate as a photonic module.Manufacturing photonic modules and electronic control circuit on thesame substrate may reduce costs because, instead of having to rely ontwo separate fabrication processes and one bonding process, only onemanufacturing process may be needed.

FIG. 3I is a cross section of a photonic substrate that co-integrates aphotonic module with transistors. The transistors may be connected toone another to define an electronic control circuit. In this example,the photonic substrate is formed on an SOI substrate, though other typesof substrates are also possible, including bulk silicon substrates. Aninsulator layer (e.g., a silicon dioxide layer) is formed on the siliconsubstrate. A silicon layer is formed on the insulator layer. The siliconlayer is patterned to form waveguides and other optical components, suchas the components described in connection with FIG. 3A. This crosssection illustrates a portion of a Mach-Zehnder interferometer havingarms defined by waveguides 370 and 371. This Mach-Zehnder interferometerdefines one of the switches of an optical distribution network 104.Transistor block 380 is formed in the same silicon layer as waveguides370 and 371. Transistor block 380 includes several transistors (e.g.,several tens of thousands, several hundreds of thousands, severalmillions, or more) connected together to form an electronic controlcircuit. The photonic substrate further includes several levels of metaltraces (although this examples shows only two levels of metal traces).Vias connect the metal traces to the waveguides and the transistors. Themetal traces enable the electronic control circuit to control theoperations of the Mach-Zehnder interferometer.

FIG. 4 is a cross sectional view of an example computing system 400based on a photonic substrate 20 (e.g., a 3×3 photonic substrate). A diestack including dies 420, 421 and 422 is mounted to the photonic modulepositioned at the left-hand side of photonic substrate 20. These diesmay form, for example, a stacked memory unit. Laser die 430 is mountedto one side of the photonic module in the middle of the substrate, anddie 431 is mounted to the opposite side of the same photonic module. Tosupport dies on opposite sides, a photonic module may include at leastone out-of-plane coupler that emits light in the upward direction and atleast one out-of-plane coupler that emits light in the downwarddirection. Dies 440 and 441 are mounted, side-by-side, to the samephotonic module. Dies 440 and 441 may include, for example, processorsor memories. As described in connection with FIG. 3D, the photonicmodules provide a platform for distributing optical signals from one dieto another.

Laser die 430 includes one or more lasers. Light produced by thelaser(s) can be distributed across the computing system and can serve asreference light to be modulated with data. Laser die 430 can includeIII-V lasers, such as InP-based lasers. Laser die 430 can be bonded tothe photonic substrate, for example using surface mounting techniques.The laser(s) of laser die 430 can be coupled to the semiconductorsubstrate using out-of-plane couplers. In some embodiments, laser lightwhich is emitted in a direction parallel to the chip surface can bedirected towards the out-of-plane couplers using ball lenses.

Recent progress in die-to-wafer bonding of III-V lasers onto a siliconphotonics-based wafer shows that the yield of the process may be wellbelow 100%. To circumvent this problem, two or more lasers can bedirected towards the same input of the photonic substrate. Only onelaser may be used at a time, but if one laser fails, then another lasercan be turned on and fed into the photonic substrate. Having multiplelasers per photonic substrate improves the reliability of the platformin case one or more lasers fail.

Chip-based III-V lasers are not the only external light source optionfor the optical communication platform. Other lasers (e.g.,semiconductor-based such as those used in optical telecommunication) canbe coupled into the platform using optical fibers or using free-spaceoptics such as lenses. In some embodiments, 1-to-N splitters can be usedto enable a single laser to provide light to multiple photonic modules,thus reducing the numbers of laser dies.

Some applications may require multiple lasers and may further requirethat optical signals emitted by the lasers be mutually coherent (e.g.,temporally coherent). In some such embodiments, optical communicationplatforms of the types described herein can be used to lock the phase ofone or more lasers to a single master laser. One or more opticaldistribution networks 105 may be configured to that the master laser ismixed with one of the slave lasers, and their beat interference patternis measured. The beat interference pattern is used as an error signal tolock the phases, so that the entire system is coherent.

Lasers emitting at different wavelengths may be used in some embodimentsto support wavelength multiplexing schemes. For example,wave-division-multiplexing (WDM) schemes can be used to increasebandwidth utilization per waveguide. Other schemes include multi-modewaveguides, time-division multiplexing and/or polarization diversity.These techniques support multiple independent communication channelsusing the same optical path.

In some embodiments, appropriate light sources other than lasers such asLEDs or superluminescent diodes may be used in lieu of theaforementioned laser. The choice of the light source is also motivatedby the choice of wavelength in the photonic communication fabric. If thefabric is intended to communicate using visible light, then the lightsources should be selected to output the appropriate wavelengths oflight.

IV. Optical Distribution Networks

Optical distribution network 104 may be implemented using opticalswitches. Examples of optical switches include Mach-Zehnderinterferometers, optical resonators, multimode interference (MMI)waveguides, arrayed waveguide gratings (AWG), thermos-optic switches,acousto-optic switches, magneto-optic switches, MEMS optical switches,non-linear optical switches, liquid crystal switches, piezoelectric beamsteering switches, grating switches, dispersive switches, etc.

Optical distribution networks 104 may be static or dynamic (e.g.,reconfigurable based on electrical or optical control signals). A staticnetwork may for example receive multiple wavelengths from the same inputwaveguide and route each wavelength to a different output waveguide.Another static network may receive two orthogonal polarizations from thesame input waveguide and route each polarization to a different outputwaveguide. Another static network may receive multiple modes from thesame input multimode waveguide and route each mode to a different outputwaveguide.

Dynamic optical distribution networks may be reconfigured depending uponthe needs of a computing system. FIG. 5A illustrates an example of adynamic optical distribution network. In this example, opticaldistribution network 104 includes a 3×1 switch 602, two 1×2 switches 606and an optical coupling element 107. This photonic module furtherincludes waveguides 111-114 and out-of-plane coupler 105, which mayenable optical communication with a laser die or other dies. Switch 602selects as input one among waveguide 111, waveguide 113 and out-of-planecoupler 105. The first switch 604 routes the input received from switch602 to either optical coupling element 107 or switch 604. Opticalcoupling element 107 directs the light power to the photonictransmitters illustrated in FIG. 6. In some embodiments, opticalcoupling element 107 includes an out-of-plane coupler for coupling lightto a die. The second switch 604 routes the input received from the firstswitch 604 to either waveguide 112 or waveguide 114. In someembodiments, switches 604 are implemented using Mach-Zehnderinterferometers.

FIG. 5B illustrates a 3×3 photonic substrate including photonic modulesof the type illustrated in FIG. 5A. In this example, the photonic chippositioned at the north-west corner of the photonic substrate is coupledto a laser 600. In some embodiments, laser 600 is formed on a laser dieand is optically coupled to out-of-plane coupler 105. In otherembodiments, laser 600 is integrated as part of that photonic module(for example, is placed in a trench formed through the top surface ofthe photonic module). Laser 600 may emit a single wavelength or multiplewavelengths. In some embodiments, laser 600 provides light to the entirephotonic substrate, though in other embodiments, other photonic modulesmay also have lasers.

FIGS. 5C and 5D illustrate other possible implementations for an opticaldistribution network 104. The example of FIG. 5C is fully-connected—allboundaries of the photonic module are mutually coupled. Light incomingfrom a boundary passes through multiple 1×2 switches 604, whichdetermine whether light proceeds straight, to the left or to the right.In some embodiments, routing may be performed for each optical channel(e.g., each waveguide mode, polarization or wavelength).

A fully connected routing topology, however, may not be necessary orfeasible in some embodiments. To reduce the complexity of the opticaldistribution network, routing may be restricted to fewer options, asshown in the example of FIG. 5D (which includes two 1×2 switches).Reducing the routing options allows for a lower number of switches peroptical module, thereby lowering power consumption and channelcrosstalk, and improving signal-to-noise ratio (SNR). These gains,however, are made at the expenses of data bandwidth.

V. Optical Interconnect Fabric

FIG. 6A illustrates an example of a reconfigurable photoniccommunication fabric. This communication fabric includes severalswitches, where “2:2” indicates a 2×2 switch and “3:3” indicates a 3×3switch. The switches may be configured depending upon the needs of acomputing system. Transceiver 700 includes optical-to-electricalconverters and electrical-to-optical converters. In some embodiments,transceiver 700 embodies optical coupling element 107 of FIG. 5A. FIG.6B illustrates an example of a transceiver 700. For clarity, only onetransmitter/receiver pair is shown, though there may be atransmitter/receiver pair for each waveguide coupled to transceiver 700.Switch 702 arbitrates between a transmitter (TX) 704 and a receiver (RX)706. TX 704 includes an electrical-to-optical converter, such as anoptical modulator. RX 706 includes an optical-to-electrical, such as anoptical receiver. FIG. 6C illustrates a 3×3 photonic substrate, whereeach node includes the photonic module of FIG. 6A.

VI. Electronic Switching Network

The inventors have appreciated that interfacing dies (e.g., memories,processors, etc.) with the photonic substrates described herein raisescompatibility issues. Ideally, dies are predefined with control circuitsfor controlling the operations of the optical distribution networks andpins that align perfectly with the pins of the photonic substrate. Inthis way, the die and the photonic substrate are inherently predisposedto communicate with each other once they are bonded together. Thisapproach, however, may be impractical because the die and the photonicsubstrate are often manufactured by different entities. For example, oneentity in the United States may manufacture the photonic substrate andanother entity in Japan may manufacture a die. This approach puts theburden on the entity that manufactures the die to include controlcircuits and pins that are compatible with the photonic substrate, whichcan substantially increase costs for the die manufacturer. Recognizingthis problem, the inventors have developed electronic switching networksthat serve as interfaces between photonic substrates and dies. Theseelectronic switching networks are arranged so that dies need not beredesigned to be compatible with a photonic substrate, thus saving coststo die manufacturers. In essence, an electronic switching networkdefines the switching and control protocols for a photonic communicationplatform.

Some electronic switching networks are co-fabricated with the photonicmodules. In some embodiments, an electronic switching network is formedon the same substrate on which the photonic modules are formed.Referring back to FIG. 3I, for example, an electronic switching networkmay be defined by transistors 380. In other embodiments, the photonicmodules are formed on a first substrate, an electronic switching networkis formed on a second substrate, and the substrates are bonded (e.g.,3D-bonded) together. Regardless of how it is formed, an electronicswitching network may be used to program the optical distributionnetworks 104. Use of electronic switching networks enables design ofdies with minimal interface circuitry, thus reducing the costs that diemanufacturers have to incur to make their dies compatible with aphotonic substrate.

FIG. 7A illustrates a photonic substrate co-integrated (e.g., bonded orformed directly on the same substrate) with an electronic switchingnetwork. This photonic substrate includes six photonic modules arrangedin a 2×3 configuration. As discussed above, each photonic moduleincludes an optical distribution network 104. The electronic switchingnetwork includes multiple controllers 740, digital-to-analog converters(DAC) 750, memories 742 and debugging units 744. Each controller 740controls an optical distribution network through a DAC 750. A controller740 may control the state of the switches of an optical distributionnetwork to dynamically reconfigure optical links as needed. Referringback to the example of FIG. 6C, a controller 740 may be used to controlthe states of the 2×2 switches and the 3×3 switches.

The electronic switching network may program the optical distributionnetworks based on data stored in the memories 742. For example, thememories may store instructions that, when executed, cause the opticaldistribution networks to perform a predefined switching sequence.Alternatively, the memories may store instructions that, when executed,cause the optical distribution networks to dynamically optimize theoptical links based on the needs of a particular application. In someembodiments, the electronic switching network may monitor usage of theoptical links, and may determine which optical links are being used andwhich optical links are available, or which optical links can provide abandwidth in excess of a threshold bandwidth. The electronic switchingnetwork may use this information to decide how to allocate optical linksto particular data streams. This may be particularly useful in photoniccommunication platforms connecting, for example, hundreds of memory diesand processors. Such photonic communication platforms can rely on theelectronic switching network to determine the optimal optical pathallowing a particular processor die to access information from aparticular memory die. Machine learning algorithms may be used in someembodiments to identify optimal optical links.

In some embodiments, an electronic switching network is configured toperform the following steps. At a first time, the electronic switchingnetwork programs the optical switches of a photonic substrate to form afirst optical communication path coupling together a first subset of theplurality of dies (e.g., coupling a first die to a second die).Programming the optical switches at the first time may involve changingthe states of some of the switches (e.g., a first subset of theswitches) without changing the states of other switches, or may involvechanging the states of all the switches. At a second time subsequent tothe first time, the electronic switching network programs the opticalswitches to form a second optical communication path coupling together asecond subset of the plurality of dies (e.g., a third die to the firstdie or a third die to a fourth die). The electronic switching networkmay continue to reprogram the optical switches as needed. Programmingthe optical switches at the second time may involve changing the statesof some of the switches (e.g., the first subset of the switches or asecond subset of the switches) without changing the states of otherswitches, or may involve changing the states of all the switches.Changing the state of a switch may involve any of the followingoperations: changing an outputs of the switch from an off-state to anon-state, changing an output of the switch from an off-state to apartially on-state (e.g., a state in which the switch delivers afraction of the input power to the output and a fraction of the inputpower to one or more other outputs), changing an output of the switchfrom an on-state to a partially on-state, changing an output of theswitch from an on-state to an off-state, selecting one or more inputs ofthe switch that were previously unselected, and/or unselecting one ormore inputs of the switch that were previously selected.

In the example of FIG. 7B, the electronic switching network hasprogrammed the photonic substrate to form an optical path from a firstdie to a second die. Subsequently, as shown in FIG. 7C, the electronicswitching network has reprogrammed the photonic substrate to form anoptical path from a third die to the first die. In some embodiments, anelectronic switching network can leverage wavelength divisionmultiplexing and/or time division multiplexing to program the opticaldistribution networks.

A typical connection between nodes may begin with a request message onthe electronic switching network. The request may be arbitrated througha mesh of electronic switches to establish a connection. At each networkhop, the optical link is made once the electrical request winsarbitration. Once the connection is fully established between the sourceand destination, an acknowledgement is returned to the requester on theelectronic network and data transmission proceeds over the optical link.

Controllers 740 may use debugging units 744 to perform testing,diagnosis and fault isolation of metal traces, contacts, pins, pads,etc. In some embodiments, the debugging units are implemented using theJoint Test Action Group (JTAG) industry standard. In some embodiments,manufacturing defects could result in non-functional communicationschannels. Redundant connections (optical and/or electrical) can beincluded and mapped in as replacements so that platforms with smallnumbers of defects need not be discarded.

The ability to dynamically reconfigure a photonic communication platformin a dynamic fashion allows the electronic switching network to operateat relatively low bandwidths, thus leading to relatively low powerconsumption. Being low power is particularly beneficial when the amountof data carried over the photonic modules between switching events isrelatively large. Being low power, in some embodiments, opens theopportunity for “always-on” operations that can monitor training orreconfiguration sequences.

In some embodiments, the power and fidelity of an optical signal thatpropagates across multiple photonic modules may depend upon the numberof waveguide crossings that the signal crosses. Thus, the farther anoptical signal has to travel the larger the power consumption and thelower the fidelity. Recognizing this issue, the inventors have developedconfigurations that can adaptively select encoding schemes with largeror smaller alphabets (more or less bits per symbol) depending upon thenumber of photonic module boundaries that a signal is supposed to crossor an expected optical loss, or any other characteristic of an opticalpath. For example, when only a few crosses or small amounts of lossesare expected, an encoding scheme with a larger alphabet can be used.Examples of such encoding schemes include quadrature amplitudemodulation (QAM) that use a constellation of 16 points (16-QAM), 64points (64-QAM), or 256 points (256-QAM), and pulse-amplitude modulation(PAM) with many levels, e.g. PAM-16 or PAM-32. Vice versa, when manycrosses or large amounts of losses are expected, an encoding scheme witha smaller alphabet can be used. Examples of such encoding schemesinclude binary phase shift keying (BPSK) or PAM with few levels, e.g.PAM-2 or PAM-4, In some embodiments, the bandwidth of an optical pathmay be set depending upon changing communication patterns, or dependingupon the number of photonic module boundaries that a signal is supposedto cross.

Clocking of the electronic switching network can be done using a singledistributed clock. The clock phase may be tuned in the photonics domainto achieve extremely low jitter at each receiver. This avoids the needfor an embedded clock and thus the data may not need to be encoded.Non-encoded data allows for increased bandwidth, reduced latency, andreduced power.

VII. Computing Systems Based on Photonic Communication Platforms

Computing systems may be formed that leverage the photonic communicationplatforms described herein. Unlike electronic distribution networks,these optical communication platforms can provide copies of the samemessage to multiple locations simultaneously without incurring problemsarising due to parasitic impedance. This characteristic allows theoptical distribution networks to form multicast and/or broadcastcommunications schemes. The optical distribution networks can bedynamically reconfigured to route messages to a single node or multiplenodes. Leveraging the broadcast and/or multicast capability, someembodiments enable performance of MapReduce operations directly using anoptical communication platform.

Computing systems of these types may be used in a variety ofapplications, including for example in high-performance computing,neural, machine learning and deep learning networks, graphics rendering,large-scale visualization, gaming, high-frequency trading and videostreaming, among others.

FIG. 8A illustrates an example computing system 800. This computingsystem is formed on a 4×4 photonic substrate. Of course, otherdimensions and topologies are also possible. In this example, there arefour processor dies 704 positioned in the middle of the photonicsubstrate and twelve memory dies 702 surrounding the processor dies.Each die is mounted to a photonic module. Each die communicates with therespective photonic module using for example out-of-plane couplers(e.g., as illustrated in FIG. 3B or FIG. 3C). Other embodiments mayinclude different numbers of processors and different numbers ofmemories, and/or may include other types of dies, including for example,analog accelerators, photonic accelerators, photonic memories,networking chips, etc. In some embodiments, computing system 800 isdisposed on an interposer, and may communicate with the interposer usingthrough-silicon vias.

The bottom inset of FIG. 8A illustrates a photonic module 722 inadditional detail. Photonic module 722 may be implemented using (orinclude) any of the photonic modules described above. For example,photonic module 722 includes an optical distribution network 104.Photonic module 722 may further include waveguides (not illustrated inFIG. 8A) allowing for optical communication with neighboring photonicmodules (see for example waveguides 111-114 of FIG. 3A). Photonic module722 further includes at least one fiber coupler 710, which may includean edge coupler and/or an out-of-plane coupler. Fiber coupler 710 can becoupled to the end of an optical fiber, thus enabling communication withother systems. Edge couplers enable optical coupling in the plane of thephotonic module. Examples of edge couplers include tapered waveguides,v-grooves and U-grooves. In some embodiments, an edge coupler includessimply the end of a waveguide at an edge of the photonic substrate. Bycontrast, out-of-plane couplers (e.g., grating couplers and prisms)enable optical coupling outside the plane of the photonic chip. Thisparticular computing system is arranged so that the lowermost photonicmodules are connected to optical fibers (as indicated by the label“fiber channels”). Photonic module 722 may further include one or moreout-of-plane couplers (not shown in FIG. 8A) enabling opticalcommunication with the die mounted to the photonic module (see forexample out-of-plane coupler 105 of FIG. 3A).

The top inset of FIG. 8A illustrates a memory die 702 in additionaldetail. Memory die 702 includes a memory block 720, which includesseveral memory units (e.g., solid state memories such as NAND, DRAM,SRAM, HBM, etc.). Memory dies 702 further includes a communication block724, which may include optical components for communicating withphotonic modules to which the memory die is mounted. For example,communication block 724 may include an out-of-plane coupler that coupleswith an out-of-plane coupler of the photonic module. Memory die 702further includes a serializer/deserializer (SERDES) block 722. SERDESblock 722 convert data from serial to parallel and vice versa. In thisparticular implementation, the SERDES block is positioned near the outeredge of memory die 702 and memory block 720 is positioned in the middleof the memory die. Of course, other arrangements are possible. AlthoughFIG. 8A does not illustrate a processor die 704 in detail, processordies may also include out-of-plane couplers that couple with theout-of-plane couplers of the respective photonic modules.

The computing systems described in connection with FIG. 8A may be usedas a stand-alone computing system, or may be used in combination withother computing systems. Combinations of computing systems are referredto herein a multi-node computing systems. FIG. 8B illustrates an exampleof a multi-node computing system including four computing systems 800.Other multi-node computing systems may include many computing systems,for example in the tens or hundreds of units, or more. Optical fibers712 are used to place the computing systems in communication with oneanother. Each end of an optical fiber couples to a fiber coupler 710 ofa computing system 800. In this example, the computing systems of themulti-node computing system share the same layout (same numbers ofphotonic modules, processor dies and memory dies). However, not allembodiments are limited in this respect. In some embodiments, thecommunication between the many multi-node computing systems can be doneelectronically by the use of silicon interposers. This communicationstrategy may consume a reasonably small amount of power if the computingsystems are placed adjacent to one another or within centimeters apartto reduce capacitance and other parasitic resistance.

The inventors have further appreciated that optical modules of the typesdescribed herein may serve as transceivers enabling communication from acomputing system to another computing system or from a computer systemto a host. This transceiver may be optical or electronic. Exampleoptical interfaces include board-to-board communication via opticalfibers or remote rack-to-rack communication utilizing higher levelprotocols such as Ethernet or Infiniband. Electronic interfaces to ahost system include SERDES-based standards such as PCI Express. Anexternal I/O module may manage communications between the host and alocal system. This includes direct memory access offload capabilitiesfor high speed data movement between remote and local memories. Theexternal I/O module may further provide local interfaces for management,calibration, boot, and reliability and serviceability (RAS).

Optical communication platforms of the types described herein canprovide layered network stacks. One example stack is configured asfollows. A physical layer includes optical interconnects includingoptical-to-electrical converters, electrical-to-optical converters, andoptical distribution networks. A datalink layer includes an electronicswitching network allowing connections between modules. In someembodiments, communication between nodes of a network may begin with arequest message on the electronic switching network. The datalink layerprocesses this request and arbitrates through a mesh of electricalswitches to establish a connection. At each photonic module boundary, anoptical link is formed if the request wins the arbitration. Once theconnection is fully established between the source and the destination,an acknowledgement may be returned to the requester on the electronicswitching network and data transmission proceeds over the photonicmodules. A transport layer involves packetization, data integrity, andbuffer allocation. The transport layer uses packetization to implementupper layer protocols on the datalink layer. Flow control may beimplemented with buffer credits. If reliability dictates additionalprotection from errors at the physical layer, data integrity includingcyclic redundancy checks (CRC) (e.g., resend) and/or forward errorcorrection (FEC) schemes can be used.

Having thus described several aspects and embodiments of the technologyof this application, it is to be appreciated that various alterations,modifications, and improvements will readily occur to those of ordinaryskill in the art. Such alterations, modifications, and improvements areintended to be within the spirit and scope of the technology describedin the application. It is, therefore, to be understood that theforegoing embodiments are presented by way of example only and that,within the scope of the appended claims and equivalents thereto,inventive embodiments may be practiced otherwise than as specificallydescribed. In addition, any combination of two or more features,systems, articles, materials, and/or methods described herein, if suchfeatures, systems, articles, materials, and/or methods are not mutuallyinconsistent, is included within the scope of the present disclosure.

Also, as described, some aspects may be embodied as one or more methods.The acts performed as part of the method may be ordered in any suitableway. Accordingly, embodiments may be constructed in which acts areperformed in an order different than illustrated, which may includeperforming some acts simultaneously, even though shown as sequentialacts in illustrative embodiments.

All definitions, as defined and used herein, should be understood tocontrol over dictionary definitions, definitions in documentsincorporated by reference, and/or ordinary meanings of the definedterms.

The indefinite articles “a” and “an,” as used herein in thespecification and in the claims, unless clearly indicated to thecontrary, should be understood to mean “at least one.”

The phrase “and/or,” as used herein in the specification and in theclaims, should be understood to mean “either or both” of the elements soconjoined, i.e., elements that are conjunctively present in some casesand disjunctively present in other cases.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified.

The terms “approximately” and “about” may be used to mean within ±20% ofa target value in some embodiments, within ±10% of a target value insome embodiments, within ±5% of a target value in some embodiments, andyet within ±2% of a target value in some embodiments. The terms“approximately” and “about” may include the target value.

What is claimed is:
 1. A photonic system comprising: a plurality ofphotonic modules including at least first and second photonic modulespatterned according to at least one common photomask, each of the firstand second photonic modules comprising: first and second boundaries; anoptical distribution network; a first optical waveguide opticallycoupling the optical distribution network to a first neighboringphotonic module of the plurality of photonic modules, the firstneighboring photonic module being adjacent to the first boundary; and asecond optical waveguide optically coupling the optical distributionnetwork to a second neighboring photonic module of the plurality ofphotonic modules, the second neighboring photonic module being adjacentto the second boundary.
 2. The photonic system of claim 1, wherein thefirst and second boundaries are opposite to one another.
 3. The photonicsystem of claim 1, wherein the first and second optical waveguides arepatterned according to the at least one common photomask.
 4. Thephotonic system of claim 1, wherein each of the first and secondphotonic modules further comprises an out-of-plane optical coupleroptically coupled to the optical distribution network.
 5. The photonicsystem of claim 1, wherein the optical distribution network isconfigured to selectively place the first neighboring photonic module inoptical communication with the second neighboring photonic module. 6.The photonic system of claim 1, wherein each of the first and secondphotonic modules is patterned according to a common set of photomasks,wherein the at least one common photomask is part of the common set ofphotomasks.
 7. The photonic system of claim 1, wherein each of the firstand second photonic modules further comprises: third and fourthboundaries, wherein the first and second boundaries are opposite to oneanother and the third and fourth boundaries are opposite to one another;a third optical waveguide optically coupling the optical distributionnetwork to a third neighboring photonic module of the plurality ofphotonic modules, the third neighboring photonic module being adjacentto the third boundary; and a fourth optical waveguide optically couplingthe optical distribution network to a fourth neighboring photonic moduleof the plurality of photonic modules, the fourth neighboring photonicmodule being adjacent to the fourth boundary.
 8. The photonic system ofclaim 7, wherein the optical distribution network is configured toselectively place the first neighboring photonic module in opticalcommunication with the second neighboring photonic module or the thirdneighboring photonic module.
 9. The photonic system of claim 1, whereinthe optical distribution network comprises a plurality of opticalswitches.
 10. The photonic system of claim 1, wherein the first andsecond photonic modules are adjacent to one another such that the secondphotonic module is the first neighboring photonic module for the firstphotonic module.
 11. A computing system comprising: a photonic substratepatterned with a plurality of photonic modules including at least firstand second photonic modules, each of the first and second photonicmodules being patterned according to at least one common photomask,wherein the first photonic module is optically coupled to the secondphotonic module; a first die in communication with the first photonicmodule; and a second die in communication with the second photonicmodule.
 12. The computing system of claim 11, wherein the first diecomprises a processor and the second die comprises a memory.
 13. Thecomputing system of claim 11, further comprising a laser die coupled tothe photonic substrate.
 14. The computing system of claim 11, whereineach of the first and second photonic modules comprises: first andsecond boundaries; an optical distribution network; a first opticalwaveguide optically coupling the optical distribution network to a firstneighboring photonic module of the plurality of photonic modules, thefirst neighboring photonic module being adjacent to the first boundary;and a second optical waveguide optically coupling the opticaldistribution network to a second neighboring photonic module of theplurality of photonic modules, the second neighboring photonic modulebeing adjacent to the second boundary.
 15. The computing system of claim14, wherein the first and second boundaries are opposite to one another.16. The computing system of claim 14, wherein the first and secondoptical waveguides are patterned according to the at least one commonphotomask.
 17. The computing system of claim 11, wherein each of thefirst and second photonic module comprises an out-of-plane opticalcoupler, wherein: the first die is optically coupled to the out-of-planeoptical coupler of the first photonic module, and the second die isoptically coupled to the out-of-plane optical coupler of the secondphotonic module.
 18. The computing system of claim 11, wherein the firstdie is coupled to a first side of the photonic substrate and the seconddie is coupled to a second side of the photonic substrate opposite thefirst side.
 19. The computing system of claim 11, further comprising athird die stacked on top of the first die.
 20. The computing system ofclaim 11, wherein the first and second photonic modules share a boundarysuch that the first photonic module is adjacent to the second photonicmodule.
 21. The computing system of claim 11, wherein: the first die ismounted above or below the first photonic module; and the second die ismounted above or below the second photonic module.
 22. The computingsystem of claim 11, wherein: the first die is in electroniccommunication with the first photonic module, and the second die is inelectronic communication with the second photonic module.
 23. A photoniccommunication platform comprising: a photonic network comprising aplurality of optical switches formed on a semiconductor substrate; aplurality of dies in communication with the photonic network; anelectronic switching network comprising a plurality of transistorsco-integrated with the plurality of optical switches, the electronicswitching network being configured to: at a first time, program theoptical switches to form a first optical communication path couplingtogether a first subset of the plurality of dies, and at a second timesubsequent to the first time, program the optical switches to form asecond optical communication path coupling together a second subset ofthe plurality of dies, the second optical communication path beingdistinct from the first communication path.
 24. The photoniccommunication platform of claim 23, wherein the plurality of transistorsare formed on the semiconductor substrate.
 25. The photoniccommunication platform of claim 23, wherein the semiconductor substrateis a first semiconductor substrate, and wherein the plurality oftransistors are formed on a second semiconductor substrate, wherein thefirst and second semiconductor substrates are 3D-bonded together. 26.The photonic communication platform of claim 23, wherein programming theoptical switches to form a first optical communication path comprises:identifying an optical communication path coupling together first subsetof the plurality of dies; and programming the optical switches based onthe identified optical communication path.
 27. The photoniccommunication platform of claim 26, wherein identifying an opticalcommunication path coupling together first subset of the plurality ofdies comprises monitoring a usage of the photonic network.
 28. Thephotonic communication platform of claim 23, wherein the electronicswitching network is further configured to: determine at least onecharacteristic of an optical signal at the first optical communicationpath; identify an encoding scheme based on the at least onecharacteristic of the optical signal; and cause the photonic network tocommunicate optically on the first optical communication path based onthe encoding scheme.
 29. The photonic communication platform of claim23, wherein the plurality of dies are in electronic communication withthe photonic network.
 30. The photonic communication platform of claim23, wherein the electronic switching network is further configured tocause the photonic network to communicate optically on the first opticalcommunication path using wavelength division multiplexing.